Image acquisition and processing systems for vehicle equipment control

ABSTRACT

The present invention provides improvements in vehicle vision system components, vehicle vision systems and vehicle equipment control systems employing the vision system components and vision systems.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. §119(e) of U.S.Provisional Patent Application Ser. No. 60/629,108, entitled “IMPROVEDIMAGE PROCESSING SYSTEM FOR VEHICLE EQUIPMENT CONTROL,” which was filedNov. 18, 2004; U.S. Provisional Patent Application Ser. No. 60/710,602,entitled “IMPROVED IMAGE ACQUISITION AND PROCESSING SYSTEM FOR VEHICLEEQUIPMENT CONTROL,” which was filed Aug. 23, 2005; and United StatesProvisional Patent Application Attorney Docket AUTO 433V3, entitled“IMPROVED IMAGE PROCESSING SYSTEM FOR VEHICLE EQUIPMENT CONTROL ANDVARIOUS VEHICLE EQUIPMENT CONTROL SYSTEMS,” which was filed Sep. 8,2005, the disclosures of which are incorporated in their entirety hereinby reference.

BACKGROUND OF THE INVENTION

It has become common to incorporate vision systems within vehicles forautomatic control and monitoring of various vehicle equipment systems.The present invention provides improvements in vehicle vision systemcomponents, vehicle vision systems and vehicle equipment control systemsemploying the vision system components and vision systems.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 depicts a plan view of a controlled vehicle;

FIG. 2 depicts an exploded, perspective, view of an exterior rearviewmirror assembly;

FIG. 3 depicts a perspective view of an interior rearview mirrorassembly;

FIG. 4 depicts a sectional, profile, view of an image sensor;

FIG. 5 depicts a sectional, profile, view of an image sensor;

FIG. 6 depicts a block diagram of a vehicle equipment control system;

FIG. 7 depicts a block diagram of a vehicle equipment control system;

FIG. 8 depicts an actual image of a scene generally in front of acontrolled vehicle;

FIG. 9 depicts a result of extracting features from the image asdepicted in FIG. 8;

FIG. 10 depicts an actual image of a scene generally in front of acontrolled vehicle;

FIG. 11 depicts a drawing of a roadway with lane markers;

FIG. 12 depicts a graph of a row of pixel data that would result from animage of the drawing of FIG. 11;

FIG. 13 depicts a graph of the first derivative of one of the lanemarkers of FIG. 12;

FIG. 14 depicts a graph of the second derivative of FIG. 13;

FIG. 15 depicts an exploded view of a section of the graph of FIG. 14;

FIG. 16 depicts the features identified in the drawing as depicted inFIG. 13;

FIG. 17 depicts a road model developed from a drawing as depicted inFIG. 13;

FIG. 18 depicts a road model developed from a drawing as depicted inFIG. 13 with a controlled vehicle superimposed; and

FIG. 19 depicts a sequence of road models developed from a sequence ofdrawings each of which as depicted in FIG. 13 with a controlled vehiclesuperimposed.

DETAIL DESCRIPTION OF THE INVENTION

Many vehicle equipment control systems have been proposed thatincorporate imaging systems and related processors. In at least oneembodiment described herein a single imaging system is provided tofacilitate multiple vehicle system functionality. In at least oneembodiment multiple imaging systems are provided to individually servemultiple or singular applications.

Vehicle exterior light control systems using a camera and imageprocessing system have been developed and disclose in commonly assignedU.S. Pat. Nos. 5,837,994, 5,990,469, 6,008,486, 6,130,448, 6,130,421,6,049,171, 6,465,963, 6,403,942, 6,587,573, 6,611,610, 6,621,616,6,631,316 and U.S. patent application Ser. Nos. 10/208,142, 09/799,310,60/404,879, 60/394,583, Ser. Nos. 10/235,476, 10/783,431, 10/777,468,09/800,460 and 60/590,736; the disclosures of which are incorporatedherein in their entireties by reference. In these systems, images areacquired of the view forward a motor vehicle. In at least oneembodiment, an image sensor is optically coupled to the interior surfaceof the windshield such that reflections and, or, refraction from theinterior windshield surface is substantially eliminated. These imagesare processed to determine the presence or absence of oncoming orpreceding vehicles and the controlled vehicles exterior lights areadjusted, for example by turning off the high beams, to prevent glare tothe drivers of other vehicles.

Moisture sensing, windshield wiper and HVAC controls are described incommonly assigned U.S. Pat. Nos. 5,923,027 and 6,617,566 as well as U.S.patent application Ser. Nos. 09/970,728 and 60/472,017, the entiredisclosures of which are incorporated herein by reference.

With reference to FIG. 1, a controlled vehicle 105 may comprise avariety of exterior lights, such as, headlight assemblies 120 a, 120 b,foul conditions lights 130 a, 130 b, front turn signal indicators 135 a,135 b, taillight assembly 125 a, 125 b, rear turn signal indicators 126a, 126 b, rear emergency flashers 127 a, 127 b, backup lights 140 a, 140b and center high mounted stop light (CHMSL) 145.

As described in detail herein, the controlled vehicle may comprise atleast one control system incorporating various components that provideshared function with other vehicle equipment. An example of one controlsystem described herein integrates various components associated withautomatic control of the reflectivity of at least one rearview mirrorelement and automatic control of at least one exterior light. Suchsystems 115 may comprise at least one image sensor within a rearviewmirror, an A-pillar 150 a, 150 b, a B-pillar 155 a, 155 b, a C-pillar160 a, 160 b, a CHMSL or elsewhere within or upon the controlledvehicle. Images acquired, or portions thereof, maybe used for automaticvehicle equipment control. The images, or portions thereof, mayalternatively, or additionally, be displayed on one or more displays. Atleast one display may be covertly positioned behind a transflective, orat least partially transmissive, electro-optic element. A commoncontroller may be configured to generate at least one mirror elementdrive signal and at least one other equipment control signal.

Turning now to FIG. 2, various components of an outside rearview mirrorassembly 210 are depicted. In at least one embodiment, an electro-opticmirror element is provided comprise a first substrate 220 having atleast one conductive/reflective coating on an inward facing surfacesecured in a spaced apart relationship with a second substrate 225having at least one conductive/reflective coating on an inward facingsurface via a primary seal 230 to form a chamber there between. In atleast one embodiment at least a portion of the primary seal is left voidto form at least one chamber fill port 235. An electro-optic medium isenclosed in the chamber and the fill port(s) are sealingly closed via aplug material 240. Preferably, the plug material is a UV curable epoxyor acrylic material. Also shown is a spectral filter material 245located near the periphery of the element. Electrical clips 250, 255 arepreferably secured to the element, respectively, via first adhesivematerial 251, 252. The element is secured to a carrier plate 260 viasecond adhesive material 265. Electrical connections from the outsiderearview mirror to other components of the controlled vehicle arepreferably made via a connecter 270. The carrier is attached to anassociated housing mount 276 via a positioner 280. Preferably, thehousing mount is engaged with a housing 275 and secured via at least onefastener 276. Preferably the housing mount comprises a swivel portionconfigured to engage a swivel mount 277. The swivel mount is preferablyconfigured to engage a vehicle mount 278 via at least one fastener 279.Additional details of these components, additional components, theirinterconnections and operation is provided herein.

Turning now to FIG. 3, there is shown an inside rearview mirror assembly310 as viewed looking at the first substrate 322 with a spectral filtermaterial 345 positioned between the viewer and a primary seal material(not shown). The mirror element is shown to be positioned within amovable housing 375 and combined with a stationary housing 377 on amounting structure 381. A first indicator 386, a second indicator 387,operator interfaces 391 and a first photo sensor 396 are positioned in achin portion 390 of the movable housing. A first information display388, a second information display 389 and a second photo sensor 397 areincorporated within the assembly such that they are behind the elementwith respect to the viewer. As described with regard to the outsiderearview mirror assembly, it is preferable to have devices 388, 389, 397at least partially covert.

In preferred embodiments of such systems, lights from other vehicles andnon-vehicular objects are identified by locating peak points ofbrightness in the image. Once located various properties of these brightpoints, such as the brightness, color, position, width, height, andmotion are determined. The values of these parameters are analyzed usingstatistical methods to determine if the bright points correspond to theheadlamps or tail lamps of other vehicles, or to non-vehicular lightsources such as signs, reflectors, or other stationary lights. Asignificant challenge in the development of the image processingalgorithms for vehicular lighting control is properly classifying thepeak points in the image. Failure to correctly identify a light sourcemay result in glare to the other vehicles, or shutting off of the highbeams at inappropriate times resulting in controlled vehicle driverdissatisfaction.

The inventors have determined that the position of the bright point inthe image is an extremely significant variable in the classification ofthe object. Peak points located in the center of the image are morelikely to correspond to vehicular light sources while sources off to theside are more likely to correspond to signs or reflectors (other factorssuch as color, brightness, and motion are preferably simultaneouslyconsidered). The inventors are also aware from experience that themanufacturing of the camera and physical mounting of a camera in avehicle is subject to variation. Thus the actual center of the image maynot be known with high precision. To alleviate these problems, factoryaim calibration is preferably utilized to establish the center of theimage in the vehicle assembly plant. Automatic continuous aimcalibration is also utilized as described in the aforementioned priorart.

While these aim methods are highly effective in establishing theappropriate image center calibration, there are limitations that thecurrent invention overcomes. An apparatus similar to one utilized forheadlamp aiming is preferably employed in the assembly plant. Anillumination source is positioned in a predetermined position in frontof each vehicle and at least one image is acquired. At least one imageis analyzed to determine if the image sensor aim is acceptable.

In at least one embodiment, the present invention improves aimingmethods by establishing an image aim calibration which occurs with everyimage cycle or with only a small number of cycles. Thus, the presentinvention is able to adapt very quickly to changes in road conditionsand establish the position of the center of the road in the image andthus determine the position of identified bright peaks in the imagerelative to the road. This information can be used to better classifythe identified peaks and results in improved performance and thepotential elimination of the need for factory aim.

In at least one embodiment of the present invention, the painted roadlane markers are identified to locate the position of the road. Theintersection of the left and right lane in the image indicates thecenter of the road. Lane departure warning systems are commerciallyavailable on vehicles which identify lane markers and warn drivers whomake lane changes without signaling. Some of these systems use an imagesensor and image processing means to identify these lanes. Thealgorithms used in these systems may be used with an exterior lightcontrol system to identify the lanes for the purpose of aiming theexterior light control system rather than, or in addition to the lanedeparture warning function. A separate lane departure warning system maybe equipment with a means to communicate the lane positions to theexterior light control system for the purpose of determining the roadposition for the exterior light control system.

A simple lane tracking algorithm is now presented which has beendetermined to be effective for lane identification for the purposedescribed herein. For this example the imaging system may be configuredas described in FIG. 4 or 5. As depicted in FIG. 4, the imaging system405 comprises an image sensor 410 mounted to a circuit board 415. Theimage sensor is encapsulated in a material 425 to form a lens assembly430 mount. The lens assembly comprises a first lens 431 configured forfocusing light rays 440 from a scene upon the image sensor. The imagingsystem further comprises a mask 445 configured to form an aperturearound the first lens. The overall image sensor resolution is 144×176pixels. As depicted in FIG. 5, the imaging system 505 comprises an imagesensor 510 mounted to a circuit board 515 with a spectral filtermaterial 520 disposed over approximately one half of the associatedpixels. The image sensor is encapsulated in a material 525 to form alens assembly 530 mount. The lens assembly comprises a first lens 531configured for focusing light rays 540 from a scene upon the half of theimage sensor such that the light rays pass through the spectral filtermaterial. The lens assembly comprises a second lens 532 configured forfocusing light rays from substantially the same scene onto the otherhalf of the image sensor such that the light rays do not pass throughthe spectral filter material. The imaging system further comprises amask 545 configured to form an aperture around the first and secondlenses. The overall image sensor resolution is 176×144 pixels. However,the array is split in two halves, each of which images substantially thesame scene but one half does so through a spectral filter. Each halfuses a subwindow of pixels, for example 144 pixels wide×50 pixels high.Preferably the unfiltered half is used for lane detection. The field ofview is preferably approximately 0.2 degrees per pixel. The lanedetection algorithm preferably operates on the lower region of theimage, for example the bottom 15 rows and does not necessarily utilizeall columns. It should be understood that the following and subsequentexamples may be applied to various image sensors with variousresolutions and various optical configurations. As costs of imagesensors and processors decrease, it may be advantageous to use an imagesensor with higher resolution and a wider field of view, for example 50degrees or more. The wider field of view will allow a larger aimcorrection, better detection of vehicles around curves, and tolerance toa wider range of windshield angles. The present invention should not beconstrued as limited to any specific type or configuration of imagesensor.

In each row processing begins from the horizontal center pixel. Movingrightwards across the row, each pixel is examined to determine if it issignificantly larger than the pixels two places to the right and left ofthe examined pixels. If so, it is determined that the pixel is imaging aportion of a bright line (i.e. the lane marker). The pixel's coordinateis stored in a list of right-lane coordinates and then the same processtakes place moving left of the center pixel. If no bright pixel isfound, then no coordinates are stored. The process repeats for each ofthe bottom 15 rows, storing the coordinates of the bright lane markerpixels in a right and left lane pixel list. If a sufficient number (forexample at least 4) of pixels were found for right or left lanes, linearregression is performed to determine the slope and intercept of a linefitting the lane points. A R² goodness of fit value is preferably usedto determine if the points fall nearly on a line and if so, theresulting linear equation is used as an indication of the lane position.

If both left and right lanes are identified with a good R² value, theposition of the lanes and road are known. The center point is computedas the intersection of these lines. If only one of the two lines isfound, the second line can be approximated by knowing the relationshipwhich exists between the slopes of the right and left lane. Thisrelationship has been experimentally determined using examples of datacollected when two lanes are present. The slopes and intercepts of onelane can be seen to generally be related to the other, since road widthsare generally consistent. Thus a reasonable approximation of the roadposition can be determined from a single lane. Once the road center andlane positions are determined, the position of an identified objectrelative to the road center can be used for an improved classification.Additionally, the position of an object relative to the lane line markercan also be used. For example objects right of the right lane are mostlikely to be signs.

In some cases road line markers will not be identified. This can becaused by a lack of paint on a rural road, snow, salt, or other sourcesof noise which obscure the lane or make it difficult for the describedalgorithm to identify the lane properly. For periods where the laneidentification is intermittent, the center from recent prioridentification of lane markers can be used. In other cases where laneshave not been identified for a longer period of time, the time averagedmean center position can be utilized. The present invention provides animprovement over prior systems by allowing the mean center to becalculated more quickly and dynamically than prior systems, due to factthat lanes are frequently visible. In cases where left and right lanesare clearly detected, the resultant center is averaged with the centerfrom other recent time computations. The mean center should only becomputed when the vehicle is traveling straight, which can be determinedfrom a vehicle yaw sensor, a steering wheel sensors, a compass, or byinsuring that the detected lanes slopes are approximately equal inmagnitude but opposite in sign, thus indicating straight travel. Whenlanes are not present, the time averaged value is used as the calibratedimage center point. FIG. 8 shows an image from the camera of a road withlane markers. FIG. 9 shows pixels selected on the lane markers using theabove described method.

In another embodiment of the present invention, the road illuminationgradient is used to determine the road position. As can be seen in FIG.8, the lane lines point to the center of the image. In FIG. 10, an imageof a snowy road is depicted; there are no visible lane markers. However,one can visually see the road and the perspective of the road narrowingto a center point. This center point is identified in software bylooking at illumination gradients. At many of the pixels in the lowerhalf of the image, there is a direction in which the brightness of thepixel relative to its neighbors changes little, and in the perpendiculardirection changes more rapidly. A direction dependent gradientcomputation filter is used to determine the magnitude and direction ofthis change. For example, the Sobel Operators: $\begin{matrix}{{\nabla f} = \begin{bmatrix}{Gx} \\{Gy}\end{bmatrix}} \\{\quad{= \quad\left\lbrack \quad\begin{matrix}{{{- 1} \cdot f_{{i - 1},{j - 1}}} - {2 \cdot f_{{i - 1},j}} - {1 \cdot f_{{i - 1},{j + 1}}} + f_{{i + 1},{j - 1}} + {2 \cdot f_{{i + 1},j}} + f_{{i + 1},{j + 1}}} \\{{{- 1} \cdot f_{{i - 1},{j - 1}}} - {2 \cdot f_{i,{j - 1}}} - {1 \cdot f_{i,{j + 1}}} + f_{{i - 1},{j + 1}} + {2 \cdot f_{i,{j + 1}}} + f_{{i + 1},{j + 1}}}\end{matrix} \right\rbrack}}\end{matrix}$

Where f_(x,y) is the pixel grayscale value of the image pixel atlocation x,y and i,j is the current pixel location at which the gradientis being computed.

From these vectors the direction of the maximum gradient is computed.The direction perpendicular to this vector will point towards the centerof the road. For any pixels exhibiting a strong gradient, theintersections of the perpendicular vectors to the gradient may becomputed. This average intersection indicates the center of the road.

Formulas other than the Sobel operators may be used to determinegradient. It is especially useful to consider pixels beyond the adjacentpixels of the examined pixel.

In at least one embodiment, the motion of detected objects may beconsidered to determine the center of the image. As described in some ofthe prior referenced commonly assigned patents and patent applications,the detected objects may be tracked over time to determine their motionvector. In general, objects tend to emanate from the center of theimage. The intersection of the motion vectors of several objectsexamined over time may be used to compute the average center point ofthe image. In cases where there are several objects this center pointmay be computed quickly.

Any of the above methods may be combined for best results. Other methodsknow in the art may also be combined with these methods. For example,when clear lanes are detected they may be used to determine the roadlocation and center. When there are no clear lanes but strong gradients,these gradients may be used. When there is no clear road identified, theroad location from recent images may be used. Finally, when the road hadnot been identified for an extended period of time, the time averagedmean center of the image from prior cycles may be used.

Classification of objects may be performed using a statistical analysisof collected and manually identified samples of objects recorded whendriving. The various parameters of the object examined may includex-position, y-position, brightness, color, width, height, age, x-motion,and y-motion. In the present invention, x-position & y-position may beexpressed as a difference from the currently identified center of theimage. The parameters are examined using statistical analysis methods,such as those in the commercially available software program Minitab.For example, a binary logistic regression may be used to develop anequation which relates these parameters to a probability that the objectis an exterior light, and another equation may be generated to determinethe probability that the object is a tail lamp.

The example data may be divided into various subsets since there is notusually a linear relationship between any of the parameters and theprobability of the object being a vehicle light. Within a subset therelationship may be more linear. Objects in the center of the image maybe analyzed to develop an equation characterizing these objects.Separate equations may be developed for different areas of the image.Separate equations may be developed for various vehicle speeds orvarious turning conditions. Turning conditions may be based upon yawrate, steering wheel sensors, compass, or derived from the roadidentification. Separate regression equations may be developed forsituations when the road center is clearly identified from situationswhere the road cannot be clearly identified. For example, when the roadis identified a strong dependence on position may be used forclassification. When the road is not identified, and the meantime-averaged image center location is utilized with a regressionequation with less dependence on position, since the positioninformation is less certain. Other methods of data analysis, such asthose described in the referenced prior art may also be used inconjunction with the methods of the present invention. The inventionsdescribed herein for identification of the road may also be used forapplication other than exterior light control, for example lanedeparture warning systems.

Image sensors and image processing systems are increasingly beingemployed to perform a wide variety safety and convenience functions inmotor vehicles. Examples of such functions include vision assistance,headlamp control, rain sensing, lane departure warning, collisionavoidance, sign recognition, and adaptive cruise control. In some cases,where the fields of view needed for the application are similar oroverlap, it is desirous to use a single camera to perform more than oneof these or other functions. A single camera will require less physicalspace and may be less expensive than using multiple dedicated cameras.

While the use of a single camera to perform multiple functions isinitially appealing, there are several technical and commercialobstacles complicating this goal. Many of the applications listed aboverequire a field of view substantially in front of the vehicle, howeverthe requirements of the camera are substantially different. A headlampcontrol system, which identifies the headlamps and tail lamps ofoncoming and preceding vehicles, requires a field of view of 30° to 50°,resolution of approximately 5-10 pixels per degree, very highintra-scene dynamic range (i.e. the ability to sense a wide variety oflight levels within a single image), very accurate color measurement forpoint light sources, and a frame rate of approximately 5 frames persecond. A lane departure warning system requires a field of view ofapproximately 25°-35°, resolution of greater than 5 degrees per pixel, awide inter-scene dynamic range to adapt to varying daytime and nighttimelight levels, and a frame rate of approximately 10 frames per second. Asign recognition system requires a narrower field of view of view but avery high resolution of greater than 20 degrees per pixel.

To perform multiple functions the processor may have to process theimage in very different ways. Reading a sign, for instance, differssubstantially in method and complexity from detecting headlamps or taillamps. Some applications can function by analyzing a continuous streamof video images. Headlamp control, in contrast, requires the imager toabruptly change between exposure times and image windows. As describedin the patents and patent applications incorporated by referenceelsewhere herein street lamps can be distinguished from headlamps bydetecting the AC ripple in their intensity. This detection requires theimager to acquire small windows at frame rates of 480 frames per second.After the streetlamp analysis, full field images are then acquired forthe next cycle.

In addition to the technical hurdles there are substantial commercialhurdles complicating implementation of multiple functions from onecamera. An automotive manufacturer may prefer to use different suppliersto provide different functionality based upon the expertise of theindividual suppliers. The image processing software and methodsdeveloped by each supplier likely utilize a wide variety of computationhardware, each optimized for the particular function. Although it may betechnically conceivable to implement several function on one processingplatform it is likely very difficult or impractical to do so. Thus, toallow several different functions to be performed with a single camerait is necessary to provide the image data to different processingplatforms provided for each application while preserving the imagesensing control flexibility needed for some of the applications tooperate properly.

The present invention provides a camera which can be controlled by oneor more of the image processing systems to allow for a variety of imageacquisition parameters while providing a continuous standard videostream to other applications.

An example embodiment of the present invention is shown in FIG. 6. Inthis example, an image sensor 601 is controlled by a processor 602.Communication of image sensor control parameters as well as image dataoccurs over communication bus 603, which may be a bi-directional serialbus, parallel bus, a combination of both, or other suitable means.Processor 602 serves to perform the headlamp control function byanalyzing the images from camera 601, determining the headlamp statebased upon these images, and communicating the determined headlamp stateto a headlamp control module 605 thru bus 604, which may be a CAN bus orany other suitable communication link.

As described in hereinabove, the headlamp control function requires theimage sensor to be activated in several different modes with differentexposure times and different readout windows. Because of thiscomplexity, Processor 602 is selected to both perform the headlampcontrol function and control the parameters of the image sensor 601.Other functions, such as those listed above, can receive the image datafrom image sensor 601 without needing the direct image sensor controlrequired by the headlamp control function. Thus, the image data fromimage sensor 601 can be communicated to one or more other processors(shown as 608, 609 & 610) from processor 602 through and image data link607. The image data link may be a MOST bus, a high-speed CAN bus, or anyother suitable electronic data communication scheme. The communicationcan be unidirectional or bi-directional. The later case allowsadditional processors to communicate with processor 602 to modify theimage acquisition parameters if required. In a preferred embodimentimage data link 607 is implemented as described in commonly assignedU.S. Patent Application publication number 20050135465, the entiredisclosure of which is incorporated herein by reference.

While performing the headlamp control function Processor 1 will requestimages of the full field of view at one or more exposure times. Theseimages will then be processed for the headlamp control function.Simultaneously with processing, these images will be sent over imagedata link 607 to the other processors. Processor 602 may perform somepre-processing such as filtering, dynamic range compression, or colorcomputation on the images before transmission. In addition to the imagedata the acquisition parameters used to take the image may be sent inthe event this information is needed by one of the other applications.Once the image data is received, the other processors may analyze thedata independent of processor 602 and perform the required function.Additional images required solely for the headlamp control function maybe acquired between transmission of images to the other processors.

During conditions when the headlamp control function is not active, suchas in daytime or when disabled, Processor 602 may still serve to acquirethe images, pre-process the images, and transmit them to the otherprocessors. Processor 602 may also perform auto-exposure control todetermine the appropriate imaging parameters for the current lightingconditions. Alternatively, processor 602 may receive instructions fromone of the other processors to adjust exposure time or other parameters.Occasionally the output from one function may be used to supplementperformance of another function. For example, the location of road lanesdetected by a lane departure warning system may be used by the headlampcontrol function to allow determination of the location of light sourcesrelative to the road location. In this case, data other than image datamay also be computed between functions over image data link 607.

In the first embodiment, Processor 602 serves as a “master” processorand the other processors serve to receive information from the master.In an alternative embodiment shown in FIG. 7 a dedicated imagecontroller 704 is provided which serves to control the image sensor 701and may serve to perform pre-processing such as auto-exposure, dynamicrange compression, filtering, or color computation. The image data isthen transmitted over data link 707 to each of the processors. Processor702 again serves to perform the headlamp control function but requestsimages from the image controller 704 rather than controlling the cameradirectly. The one or more additional processors 708 & 709 may alsorequest specific image data from image controller 704 or may simplyreceive image data on a regular interval. Image controller 704 managesthe image requests from multiple processors while providing a regularoutput of image data to all processors. It is envisioned that imagecontroller 704 may be provided integral with image sensor 701 andpossibly even integrated monolithically on the same silicon chip as theimage sensor.

In both embodiments described herein the image sensor 701 may be locatedon the mount of a vehicle rear-view mirror. Locating the camera on themirror mount has several advantages: The mount is rigid and stationary,the mirror mount is typically located in the vehicle's windshield wiperpath, and the factory installation is simplified as the mirror isalready being attached. The camera may be placed separate from themirror, but an additional factory installation step is then required.

Regardless of the location of image sensor 701, processor 702 (oralternatively image controller 704) may be co-located with image sensor701, on the same or separate circuit boards. These processors may alsobe located in a rear-view mirror body and may serve to perform otherfunctions such as a compass sensor or control of an auto-dimmingrear-view mirror. These processors may also be located in a headliner,over-head counsel, or other suitable location in the vehicle.

Turning now to FIG. 11 an image of a roadway is depicted including leftlane line 1105, center lane line 1110 and right lane line 1115. In apreferred embodiment, the sensitivity of the associated image sensor isset such that the area within the image void of lane lines results inrelated pixel values of approximately twenty percent of the full scalevalue obtainable from the given pixels, it should be understood that thesensitivity may be set to result in thirty percent, forty percent, fiftypercent or any other desired value. The most preferred sensitivitysetting will result in the pixels actually detecting lane markingshaving a value less than full scale (i.e. not washed out).

FIG. 12 depicts a graph of the pixel values of a representative row ofthe image of FIG. 11. The left lane line 1205, the center lane line 1210and the right lane line 1215 induce higher values in the associatedpixels. It is desirable to identify the pixels in each row of the imagethat correspond to the edges of the given lane line. In a preferredembodiment a first derivative is taken of the values as depicted in FIG.12; the result of the left lane line is depicted in FIG. 13 as firstderivative 1305. Taking the first derivative of the values of FIG. 12results in “thresholding out” noise associated with the raw pixelvalues. In even a more preferred embodiment a second derivative 1405 iscalculated resulting in the graph depicted in FIG. 14. The secondderivative reveals a positive to negative transition between points 1406and 1407 indicative of a first edge of a lane line and a negative topositive transition between points 1409 and 1408 indicative of a secondedge of a lane line. Taking the second derivative results inidentification of the point of inflection associated with the given rowof pixel values being analyzed. FIG. 15 depicts an exploded view of thepositive to negative transition 1505 with point 1506 corresponding to afirst pixel and point 1507 corresponding to a second pixel;interpolation of these values results in determining a precise location1508 for an edge of the associated lane line. It should be understoodthat similar analysis may be performed to precisely locate each edge ofeach lane line within the associated image.

Turning now to FIG. 16 a translated image is depicted to include a firstfeature 1605, a second feature 1610 and a third feature 1615. In anideal situation these three features will correspond to the left, centerand right lane lines of the original image with associated noise removedor reduced as compared to the original image pixel values.

In a preferred embodiment, the values of FIG. 16 are transposed toderive a “plan view” of the corresponding left line 1705, center line1710 and right line 1715. As depicted in FIG. 18 a rectangle 1820indicative of the controlled vehicle is combined with a horizontal line1825 to be used to determine whether or not the controlled vehicle isdeviating from the appropriate lane. If the vehicle is suppose to betraveling in the right lane a determination will be made to check forintersection of either line 1810 or 1815 with the horizontal line 1825and or rectangle 1820. If the vehicle is suppose to be traveling in theleft lane a determination will be made to check for intersection ofeither line 1805 or 1810 with the horizontal line 1825 and or rectangle1820. If either of the associated lines is found to be intersecting withthe horizontal line 1825 and or rectangle 1820 an audible and or visualalarm may be initiated within the controlled vehicle cabin to alert thedriver of a lane departure. It should be understood that an appropriateaudible and or visual alarm device may be incorporated into a rearviewassembly along with at least one corresponding image sensor and or atleast one processor. It should also be understood that an output of agiven processor and or image sensor may be provided to an originalequipment manufacture to initiate an audible and or visual alarmanywhere within the vehicle in sight or hearing range of the driver. Itshould be understood that automatic steering may also be configured tobe activated as a result of the lane detection algorithm discussedabove. The steering of the vehicle may be “encouraged” to guide thecontrolled vehicle in a certain direction which may be overcome by thedriver with slightly more force than required to steer the vehiclewithout a lane departure detected.

Turning to FIG. 19 a lane departure detection algorithm is describedwith reference to three consecutively acquired images. A first imagedepicted with solid graphics includes a left lane line 1905 a, a centerlane line 1910 a, a right lane line 1915 a and a controlled vehicle 1920a. A second image depicted with dotted graphics includes a left laneline 1905 b, a center lane line 1910 b, a right lane line 1915 b and acontrolled vehicle 1920 b. A third image depicted with dashed graphicsincludes a left lane line 1905 c, a center lane line 1910 c, a rightlane line 1915 c and a controlled vehicle 1920 c. In a preferredembodiment a controlled vehicle yaw sensor input and or a controlledvehicle speed input are combined with a sequence of consecutivelyacquired images to determine when the controlled vehicle has crossed oris about to cross a given lane line. As described with regard to theabove embodiment different lane lines will be analyzed depending whetherthe controlled vehicle is suppose to traveling in the right lane or leftlane. In a preferred embodiment, the speed of the controlled vehicle,the yaw of the controlled vehicle and the consecutively acquired imagesare combined to anticipate a lane departure. In a preferred embodimentan audible and or visual alarm is initiated upon an impending lanedeparture. In at least one embodiment the controlled vehicle steering iseffected as described above.

In at least one embodiment at least one expected line width shall beutilized in determining whether a given “feature” is actually a laneline of interest or non-lane line noise. For example, an expected linewidth may be compared to an actual line width at a given distance fromthe controlled vehicle and the algorithm will perform a specificsubroutine of subroutines based upon the difference from the expectedwidth compared to the actual width. In at least one embodiment an imagesensor assembly is configured such that an expected line width atapproximately ten meters from the controlled vehicle is approximatelyfour pixels wide; it should be understood that from three to four pixelswide at approximately ten meters is preferred. In at least oneembodiment the expected lane line width is greater than one pixel attwenty-five meters. The width of the line may be determined as describedelsewhere herein. In at least one embodiment an expected lane line pixelwidth, an expected lane line, a sub-combination thereof or a combinationthereof are utilized to fix a position of a given feature relative theposition of a controlled vehicle. It should be understood that givenfeature's characterization as being a lane line may be inferred fromgeographical dependent expected data. Such as for example having alookup table of lane line widths dependent upon geographical dataautomatically selected based upon a geographical positioning system(GPS) incorporated into the controlled vehicle. It should be apparentthat lane width for inference of a second feature based upon finding afirst may also be stored in a geographically dependent lookup table. Itshould be understood that road dependent systems, such as magnets, ormagnetic material, strategically placed periodically along a roadway maybe incorporated as they become more available. As GPS data becomes moreprecise and reliable that information may be used in combination withgeographically dependent empirical data regarding the environment inwhich the controlled vehicle is traveling. The geographically dependentand visually dependent systems may be configured to enhance performanceof the individually employed technologies.

In at least one embodiment an additional feature, not identified in agiven image or given images, may be inferred from an expected featuregiven the fact that at least one other feature was found in a givenimage or a recent preceding image. In at least one embodiment the systemis configured such lane lines are expect to be a predetermined distancefrom one another, therefore, position of a second lane line may beinferred from detection of the position of a first. Many nuisancesituations such as at least partially snow covered roads, at leastpartially wet roads, at least partially shaded roads, road markingsaside from lane lines, tar strips, skid marks of other tires, paintedarrows and the like in the road and construction markings may beexpected. In at least one embodiment various expected “featurecharacteristics” are utilized to distinguish actual lane lines fromnuisances. Many of the techniques taught herein are valuable for thatpurpose.

In at least one embodiment pixel values extracted from at least oneimage are divided into a plurality of cells defining a series ofsub-windows within the original image. These individual cells aresubsequently analyzed to identify lane markers within each. Featuresextracted from the individual cells are then reassembled in a roadmodel. One advantage of utilizing cells is to account for variations inthe scene due to, for example, shadows cast on the roadway frombuildings, trees, bridges and the like. Additionally, variations inpavement and/or road surfaces within a given image may be accounted for.As an example, an image may be divided into a series of three-by-threecells. It should be understood that an image may alternatively bedivided into two columns, two rows, four columns, four rows, anysub-combination thereof or combination thereof. More cells may beemployed within the spirit of the present invention.

Whether a complete image or a cell is being analyzed, in at least oneembodiment the analysis begins by computing a running average of two,three, four, five or more pixel values across a given row. This step inthe analysis will eliminate localized points of inflection in thepursuing analysis. In at least one embodiment, any pixel values in thegiven row below an overall row average are assigned a value equal to therow average. This procedure reduces the contrast in the resulting data.In at least one embodiment a first derivative is computed across therow. Subsequent to computing the first derivative, in at least oneembodiment a group of first derivative values are utilized to compute anaverage and/or a middle range. In at least one embodiment the smoothedfirst derivative data is then utilized to compute a second derivative.In at least one embodiment the second derivative data is utilized toidentify lane markers by identifying associated rising and fallingedges. In at least one embodiment the above analysis is employed toaccurately detect lane markers on wet roadway surfaces, roadway surfacespartially illuminated from other cars and or roadway lighting.

In at least one embodiment when a group of pixels in a given row or dataare determined to be indicative of a “wide” bright spot, for examplemore than what would be expected for a lane marker, the data associatedwith a column defined by the wide bright spot is ignored in theanalysis. This analysis is particularly well suited for dealing withillumination from oncoming vehicles at night or during dark, rainy,conditions.

In at least one embodiment a series of images are analyzed to detectlane markers. If a lane marker is determined to be present in one imageand again in the next image a counter is incremented. If a lane markeris not detected in a subsequent image the counter is decremented. Oncethe counter reaches a predetermined threshold number the presents of alane marker is determined to be verified. This analysis provides ahigher degree of certainty as to detection of lane markings.

In at least one embodiment raw data from an image is first averaged andpixel values below the average are assigned the average value.Subsequently a first derivative is calculated. A thresholding functionutilizing a histogram of this data is derived then weighted. Values inthe histogram below 0.33 of the histogram are then disregarded and thevalues are assigned a zero value. A second derivative is thencalculated. Finally, the points of inflection of the second derivativeare utilized to interpolate zero crossing values.

It should be understood that the above description and the accompanyingfigures are for illustrative purposes and should in no way be construedas limiting the invention to the particular embodiments shown anddescribed. The appending claims shall be construed to include allequivalents within the scope of the doctrine of equivalents andapplicable patent laws and rules.

1. An apparatus, comprising: at least one image sensor; and at least oneprocessor configured to receive at least a portion of at least one imagefrom said at least one image sensor, wherein an aim of said at least oneimage sensor is a function of said at least one image; said processor isfurther configured to generate at least one vehicle equipment controlsignal as a function of at least one feature extracted from at least aportion of at least one image.
 2. An apparatus as in claim 1 configuredto detect at least one lane marker.
 3. An apparatus as in claim 2wherein an aim of said image sensor is a function of said at least onelane marker.
 4. An apparatus as in claim 1 configured as a rearviewmirror assembly comprising a stationary housing, said stationary housingcomprising at least one device selected from the group comprising: asecond imager sensor, an automatic exterior light control module, amoisture sensor module, a compass sensor, a compass, a speaker, amicrophone, a windshield wiper automatic control, a digital signalprocessor, an automatic defogger control, a collision avoidance control,a lane departure warning module, an electro-optic mirror element controlmodule, a supplemental illuminator module, a photo sensor and a secondprocessor.
 5. An apparatus as in claim 1 configured as a rearview mirrorassembly comprising, said rearview mirror assembly comprising at leastone device selected from the group comprising: an imager, an automaticexterior light control module, a moisture sensor module, a compasssensor, a compass, a speaker, a microphone, a windshield wiper automaticcontrol, a digital signal processor, an automatic defogger control, acollision avoidance control, a lane departure warning module, anelectro-optic mirror element control module, a supplemental illuminatormodule, a photo sensor and a second processor.
 6. An apparatus as inclaim 1 configured to function as at least one of the group selectedfrom: an automatic exterior light control system, a moisture sensingsystem, a windshield wiper control, a defroster control, a defoggercontrol, a lane keeping system, a lane departure warning system, asecurity system, a vision system, an adaptive cruise control system, aparking aid system, a blind spot warning system, a sun load sensingsystem, a blue sky detection system, a tunnel detection system, a daytime running lights control system, a security system, an air bagactivation system, a rear vision system, an occupancy detection system,a monitoring system, a collision avoidance system, an accidentrecreation system and an image acquisition system.
 7. An apparatus,comprising: at least one image sensor; a first processor configured toreceive at least a portion of at least one image from said at least oneimage sensor, wherein an aim of said at least one image sensor is afunction of said at least a portion of at least one image; and a secondprocessor configured to generate at least one vehicle equipment controlsignal as a function of at least a portion of at least one image.
 8. Anapparatus as in claim 7 configured to detect at least one lane marker.9. An apparatus as in claim 8 wherein an aim of said image sensor is afunction of said at least one lane marker.
 10. An apparatus as in claim7 configured as a rearview mirror assembly comprising a stationaryhousing, said stationary housing comprising at least one device selectedfrom the group comprising: a second image sensor, an automatic exteriorlight control module, a moisture sensor module, a compass sensor, acompass, a speaker, a microphone, a windshield wiper automatic control,a digital signal processor, an automatic defogger control, a collisionavoidance control, a lane departure warning module, an electro-opticmirror element control module, a supplemental illuminator module, aphoto sensor and a third processor.
 11. An apparatus as in claim 7configured as a rearview mirror assembly comprising, said rearviewmirror assembly comprising at least one device selected from the groupcomprising: an imager, an automatic exterior light control module, amoisture sensor module, a compass sensor, a compass, a speaker, amicrophone, a windshield wiper automatic control, a digital signalprocessor, an automatic defogger control, a collision avoidance control,a lane departure warning module, an electro-optic mirror element controlmodule, a supplemental illuminator module, a photo sensor and a thirdprocessor.
 12. An apparatus as in claim 7 configured to function as atleast one of the group selected from: an automatic exterior lightcontrol system, a moisture sensing system, a windshield wiper control, adefroster control, a defogger control, a lane keeping system, a lanedeparture warning system, a security system, a vision system, anadaptive cruise control system, a parking aid system, a blind spotwarning system, a sun load sensing system, a blue sky detection system,a tunnel detection system, a day time running lights control system, asecurity system, an air bag activation system, a rear vision system, anoccupancy detection system, a monitoring system, a collision avoidancesystem, an accident recreation system and an image acquisition system.13. An apparatus, comprising: at least one image sensor; and at leastone processor configured to receive at least a portion of at least oneimage from said at least one image sensor, wherein an aim of said atleast one image sensor is a function of said at least one image; saidprocessor is further configured to detect at least one lane marker as afunction of at least a portion of at least one image.
 14. An apparatusas in claim 13 wherein an aim of said image sensor is a function of saidat least one lane marker.
 15. An apparatus as in claim 13 configured asa rearview mirror assembly comprising a stationary housing, saidstationary housing comprising at least one device selected from thegroup comprising: an imager, an automatic exterior light control module,a moisture sensor module, a compass sensor, a compass, a speaker, amicrophone, a windshield wiper automatic control, a digital signalprocessor, an automatic defogger control, a collision avoidance control,a lane departure warning module, an electro-optic mirror element controlmodule, a supplemental illuminator module, a photo sensor and aprocessor.
 16. An apparatus as in claim 13 configured as a rearviewmirror assembly comprising, said rearview mirror assembly comprising atleast one device selected from the group comprising: an imager, anautomatic exterior light control module, a moisture sensor module, acompass sensor, a compass, a speaker, a microphone, a windshield wiperautomatic control, a digital signal processor, an automatic defoggercontrol, a collision avoidance control, a lane departure warning module,an electro-optic mirror element control module, a supplementalilluminator module, a photo sensor and a processor.
 17. An apparatus asin claim 13 configured to function as at least one of the group selectedfrom: an automatic exterior light control system, a moisture sensingsystem, a windshield wiper control, a defroster control, a defoggercontrol, a lane keeping system, a lane departure warning system, asecurity system, a vision system, an adaptive cruise control system, aparking aid system, a blind spot warning system, a sun load sensingsystem, a blue sky detection system, a tunnel detection system, a daytime running lights control system, a security system, an air bagactivation system, a rear vision system, an occupancy detection system,a monitoring system, a collision avoidance system, an accidentrecreation system and an image acquisition system.
 18. An apparatus,comprising: at least one image sensor; and at least one processorconfigured to receive at least a portion of at least one image from saidat least a portion of at least one image sensor, said processor isconfigured to compute an average of at least one row of said at least aportion of at least one image, assign the average value to any pixelvalues below said average, take a first derivative of modified pixeldata, threshold the modified pixel data via a histogram analysis andthen take a second derivative of threshold modified pixel data.
 19. Anapparatus as in claim 18 configured as a rearview mirror assemblycomprising a stationary housing, said stationary housing comprising atleast one device selected from the group comprising: an imager, anautomatic exterior light control module, a moisture sensor module, acompass sensor, a compass, a speaker, a microphone, a windshield wiperautomatic control, a digital signal processor, an automatic defoggercontrol, a collision avoidance control, a lane departure warning module,an electro-optic mirror element control module, a supplementalilluminator module, a photo sensor and a processor.
 20. An apparatus asin claim 18 configured as a rearview mirror assembly comprising, saidrearview mirror assembly comprising at least one device selected fromthe group comprising: an imager, an automatic exterior light controlmodule, a moisture sensor module, a compass sensor, a compass, aspeaker, a microphone, a windshield wiper automatic control, a digitalsignal processor, an automatic defogger control, a collision avoidancecontrol, a lane departure warning module, an electro-optic mirrorelement control module, a supplemental illuminator module, a photosensor and a processor.
 21. An apparatus as in claim 18 configured tofunction as at least one of the group selected from: an automaticexterior light control system, a moisture sensing system, a windshieldwiper control, a defroster control, a defogger control, a lane keepingsystem, a lane departure warning system, a security system, a visionsystem, an adaptive cruise control system, a parking aid system, a blindspot warning system, a sun load sensing system, a blue sky detectionsystem, a tunnel detection system, a day time running lights controlsystem, a security system, an air bag activation system, a rear visionsystem, an occupancy detection system, a monitoring system, a collisionavoidance system, an accident recreation system and an image acquisitionsystem.
 22. An apparatus, comprising: at least one image sensor; a firstprocessor configured to receive at least a portion of at least one imagefrom said at least one image sensor, wherein an aim of said image sensoris a function of said at least one image; and a second processorconfigured to detect at least one lane marker as a function of at leasta portion of at least one image.
 23. An apparatus as in claim 22 whereinan aim of said image sensor is a function of said at least one lanemarker.
 24. An apparatus as in claim 22 configured as a rearview mirrorassembly comprising a stationary housing, said stationary housingcomprising at least one device selected from the group comprising: animager, an automatic exterior light control module, a moisture sensormodule, a compass sensor, a compass, a speaker, a microphone, awindshield wiper automatic control, a digital signal processor, anautomatic defogger control, a collision avoidance control, a lanedeparture warning module, an electro-optic mirror element controlmodule, a supplemental illuminator module, a photo sensor and aprocessor.
 25. An apparatus as in claim 22 configured as a rearviewmirror assembly comprising, said rearview mirror assembly comprising atleast one device selected from the group comprising: an imager, anautomatic exterior light control module, a moisture sensor module, acompass sensor, a compass, a speaker, a microphone, a windshield wiperautomatic control, a digital signal processor, an automatic defoggercontrol, a collision avoidance control, a lane departure warning module,an electro-optic mirror element control module, a supplementalilluminator module, a photo sensor and a processor.
 26. An apparatus asin claim 22 configured to function as at least one of the group selectedfrom: an automatic exterior light control system, a moisture sensingsystem, a windshield wiper control, a defroster control, a defoggercontrol, a lane keeping system, a lane departure warning system, asecurity system, a vision system, an adaptive cruise control system, aparking aid system, a blind spot warning system, a sun load sensingsystem, a blue sky detection system, a tunnel detection system, a daytime running lights control system, a security system, an air bagactivation system, a rear vision system, an occupancy detection system,a monitoring system, a collision avoidance system, an accidentrecreation system and an image acquisition system.